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ABSTRACT 



The Assessment Tools for Adult Education project was 
designed to provide training and support to staff of the Pennsylvania Bureau 
of Adult Basic and Literacy Education (T^LE) funded programs to help them use 
assessment tools and procedures to document the learning gains of the adult 
students they serve. The following candidate assessment instruments for 
reporting learning gains were pilot tested: (1) the Woodcock- Johnson Revised 

Tests of Achievement and the Woodcock Reading Mastery Tests -Revised; (2) the 
Wonderlic Basic Skills Test; and (3) the Comprehensive Adult Student 
Assessment System (CASAS) . All three assessment instruments were deemed 
usable by a variety of ABLE- funded programs and were added to Pennsylvania's 
list of standardized assessments approved for reporting adult learner 
educational gain data. A total of 68 participants from 38 state agencies 
received CASAS initial training, and 37 participants from 17 agencies 
received CASAS advanced training. Information about and technical assistance 
on the CASAS was provided to ABLE agencies throughout the project year. Four 
Pennsylvania-based CASAS trainers were advanced toward becoming certified 
state- level CASAS trainers. (The following items are appended: the pilot test 
application form; a list of agencies participating in the pilot test; and 
draft educational gain standards for the pilot instruments.) (MN) 
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Abstract 



Contract Number: ()98-00-0()10 



Grant Recipient: 

Tiiscarora Intermediate Unit 1 1, Adult Education and Job Training Center 

MCIDC Plaza, Building 58 

6395 SR 103 North 

Lewistovvn, PA 1 7044 

(717) 248-4942 

Program Name: Assessment Tools for Adult Education 
Grant Allocation: 575,000 
Project Period: July 1, 1999 - September 30, 2000 
Project Director: Carol Shefrin 

Project Purpose: This project provided training and support to staff of ABLE-funded 
programs to help them utilize assessment tools and procedures to effectively document the 
learning gains of the adult students they serve. 

Project Outcomes: The project 1) pilot tested three candidate assessment instruments for 
reporting of learning gains, 2) expanded training and technical assistance opportunities for the 
Comprehensive Adult Student Assessment System (CASAS) through the Training Module 
Project, and 3) continued development of state-based CASAS trainers through the Training 
Module Project. 

Impact: Results of the pilot test revealed that the three assessment instruments (Woodcock 
Reading Mastery Tests-Revised, the Wonderlic Basic Skills Test, and the Secondary Level 
English Proficiency Test) were usable by a variety of ABLE-funded programs and useful for 
reporting learning gains of adult students. In response to the pilot test, the Bureau of ABLE 
added these three instruments to the list of standardized assessments approved for reporting 
adult learner educational gain data. Thirty-eight agencies (68 participants) received CASAS 
Initial Training; 1 7 agencies (37 participants) received CASAS Advanced Training. 
Information about and technical assistance on CASAS was provided to ABLE agencies 
throughout the project year. Four Pennsylvania-based CASAS trainers were advanced 
toward becoming certified state-level CASAS trainers for Pennsylvania. 

Products or Training Developed: A final report summarizes project activities, results, and 
impact. 

Products Available From: the AdvancE State Literacy Resource Center 

Project Continuation and/or Future Implications: The project provided a model for how 
additional candidate instruments may be pilot-tested for use by ABLE agencies in the future. 
CASAS training was smoothly integrated into an existing statewide delivery structure 
provided by the Training Module Project. 

Conclusions/Recommendations: Awareness-building and training on the new instruments 
would be useful to ABLE-funded agencies. Both Initial and Advanced CASAS training 
should continue to be made available to ABLE program staff. 

Additional Comments: 
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Introduction 



Purpose and Objectives of the Project 

“Assessment Tools for Adult Education” continued, coordinated, and 
expanded two of Pennsylvania's Bureau of Adult Basic and Literacy Education 
(ABLE) funded assessment training and support initiatives operating in Pennsyhania 
during previous program years. The goal of the project was to increase opportunities 
for training and teclmical assistance on assessment issues to staff of ABLE-funded 
programs in order to help them engage in continuous quality improvement and meet 
the requirements of the Workforce Investment Act. The focus of the project was to 
provide training and support to staff of ABLE-funded programs to help them utilize 
assessment tools and procedures to effectively document learning gains of the adult 
students they serve. The original project objectives were: 

1 . To adapt and pilot test appropriate assessment procedures and an Individual 
Education Plan (lEP) suitable for use with short-term learners in three programs; 

2. To identify assessment tools and develop training on those tools, as appropriate, 
based on the results of the ’98 — ’99 “ABLE Assessment Practices” project findings 
and needs resulting from the establishment of the CareerLink system; 

3. To implement training and provide technical assistance on appropriate assessment 
tools; 

4. To provide training, support, and technical assistance, and capacity-building for 
use of the Comprehensive Adult Student Assessment System (CASAS) by ABLE 
providers; 
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5. To coordinate training implementation and provision of technical assistance with 
the Training Module Project. 

Rationale for and Background of the Project 

During the ‘97 - ’98 and ’98 - ’99 program years, the Tuscarora Intermediate 
Unit’s Adult Education and Job Training Center (AEJTC) implemented the Training 
Module Project. This project developed several training modules designed to increase 
program staffs knowledge of basic assessment concepts and how to use standardized 
assessments and their resulting data to document learning gains of adult students. 
These modules included a basic 12-hour “Assessment for Adult Learners” module, as 
well as a “Hands-on Session with the TABE” module and a “Hands-on Session with 
the BEST” module (each approximately 3 hours). The “Assessment for Adult 
Learners” module provides an opportunity for participants to review and work several 
Bureau-recommended standardized assessments, while the “hands-on” sessions 
provide participants with an in-depth look at and step-by-step guidelines for 
administering, scoring, interpreting, and reporting results of the instrument of focus. 
The project also established a system for delivery of these training modules through 
Pennsylvania’s six regional ABLE professional development centers and trained a 
group of experienced and knowledgeable assessment trainers who continue to deliver 
the modules throughout the six regions. 

Another project completed by the AEJTC in the ’98 - ’99 program year 
(“ABLE Assessment Practices”) researched the assessment instruments that were 
being used by ABLE-funded providers for reporting student learning gains to the 
Bureau of ABLE and recommended a short list of standardized instruments 
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appropriate for tracking gains of learners in ABLE programs. This project rev ealed 
that the majority of ABLE-funded programs were finding this short list oi 
standardized instruments adequate for at least some instructional planning as well as 
reporting purposes. However, the project documented the need for instiuments that 
would better detect instructional needs and educational gains of three sub-populations 
of ABLE learners: 

* Beginning-level adult readers; 

* Adult learners focusing on a specific career area (e.g. nursing) rather than on 
general, employment-related literacy skills; and 

* Upper-level ESL learners who have progressed beyond the functional level 
measured by the Basic English Skills Test (BEST). 

The project collected information and reviewed standardized instruments potentially 
appropriate for these areas of need, and provided recommendations for instruments 
that might meet these needs. These included: 

* The Woodcock-Johnson Revised Tests of Achievement and the Woodcock Reading 
Mastery Tests-Revised, beginning-level adult readers; 

* The Wonderlic Basic Skills Test, career-focus learners; 

* The CASAS Employability Competency and Life Skills Systems (not widely used 
by Pennsylvania ABLE practitioners at the time) and the Secondary Level English 

Proficiency Test (SLEP), upper-level ESL learners. 

A third AEJTC project funded in ’98 - ’99 (“Project Learner.Goals”) explored 
the assessment and instructional needs of short-term or project learners w ho come 
to ABLE agencies with specific short-term learning goals in mind (e.g., preparation to 
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take the commercial driver's license exam). These learners may quickly meet their 
instructional noals and leave the program. Because of their short time within 
programs and highly-focused goals, standardized testing is not appropriate for 
documenting their learning gains. Thus, this research explored how agencies can 
document service to these learners, developed an Individual Education Plan (lEP) to 
provide the Bureau of ABLE with information on students engaging in pro ject 
learning, and identified three structured approaches to informal assessment that may 
work with project learners, including; 

* Goal Attainment Scaling (GAS); 

* Oregon's Assessment, Instruction, and Mastery (AIM); and 

* Informal assessment procedures being developed by the National Institute for 
Literacy’s “Equipped for the Future” initiative. 

A fourth training initiative canned out by the AEJTC in the '97 - 98 and 98 
- ’99 project years was statewide training in the Comprehensive Adult Student 
Assessment System (CASAS) Employability Competency and Life Skills Systems. 

As part of this training initiative, ABLE-funded agencies participated in a basic level 
training that introduced the two systems and provided information needed to purchase 
and use the systems within ABLE programs. A level-two training was offered to 
agencies that chose to adopt CASAS and needed additional support in 
implementation. In addition, regional trainers-in-training participated in an 
apprenticeship training process for certification as state-level CASAS trainers. 

The current project proposed to continue and coordinate these various 
initiatives by developing training for appropriate new standardized instruments (with 
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ihe support and approval of the Bureau of AB1..F.) through the I'raining Modulo 
Project; expanding training opportunities for CASAS and continuing to develop state- 
based CASAS trainers through the Training Module Project; and continuing to 
research potential techniques assessment and instruction of short-term “project 
learners” by adapting and pilot-testing the lEP and assessment techniques in agencies 
serving such learners. 

Change of Project Focus 

Two major changes were made in the project scope of work during the 
program year. 

First, the project staff learned in Spring, 2000, after the proposal tor this 
project was submitted, that the new federal National Reporting System (NRS) 
guidelines, scheduled to go into effect on July 1, 2000, did not provide a category for 
short-term or project learners other than “work-based project learners.” The NRS 
defines the work-based project learner as one w'ho participates in a short-term (12 - 
30 hours of duration) instructional program designed to teach specific work-based 
skills. In light of the orientation of the NRS toward project learners, the “Framework 
for Work-based Foundation Skills” and “PA Win” projects, being conducted by the 
Institute for the Study of Adult Literacy in the ’99 - ’00 project year, seemed to 
provide a better foundation for continued work with assessment and instruction of 
project learners. In May, 1999, the AEJTC project staff, the staff of the Institute for 
the Study of Adult Literacy associated with the “PA Win” and “Work-based 
Foundation Skills” projects, and Bureau of ABLE staff met and determined to 
continue the work with assessment and instruction of work-based project learners 
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under the auspices of the two Institute for the Study oT Adult Literacy projects. Since 
the AEJTC staff involved in the project were also on the advisory team to the ”P.A 
Win” and “Work-based Foundation Skills” projects in the ’99 - ’00 project year, the 
transfer of this work to the Institute’s projects would be seamless. 

Second, at the first meeting of the project staff and Cheryl Keenan, Director of 
the Bureau of ABLE, in August, 1999, the group decided to use the program year to 
pilot-test three new instruments identified through the “ABLE Assessment Practices” 
project (SLEP, Woodcock Reading Mastery Tests-Revised, and Wonderlic Basic 
Skills Test) in ABLE programs before adopting the instruments for learner gain data 
reporting or developing training on the instruments. The group decided that any 
decision on whether to adopt any or all of the instruments for learner gain data 
reporting, and any plans to develop training on administration, scoring, interpreting, 
and reporting of the instruments, would be deferred until after the completion of the 
pilot test in the ’99 - ’00 project year. A fourth instrument. Work Keys, was added to 
the list of instruments to pilot test. The Work Keys assessments had not been 
reviewed as part of the ABLE Assessment Practices research project; because the 
tests are highly secure, the researcher was unable to obtain actual samples of the tests 
or detailed technical information about them. However, the Bureau of ABLE located 
an independent source of information indicating that the Work Keys assessments met 
the criteria established by the research project for being identified as a standardized 
instrument appropriate for tracking gains of learners in ABLE programs. The project 
team believed that the Work Keys assessments might be beneficial to ABLE agencies 
that were beginning to work with their local CareerLinks. 
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Thus, ihe first project objective (To adapt and pilot tost appropriate 
assessment procedures and an Individual Education Plan (lEP) suitable tor use with 
short term learners in three programs) and third project objective (To implement 
training and provide technical assistance on newly-identified appropriate assessment 
tools) were deleted from the scope of work. The second project objective (To 
identify assessment tools and develop training on those tools, as appropriate, based on 
the results of the ’98 - ’99 “ABLE Assessment Practices” project findings and needs 
resulting form the establishment of the CareerLink system) was modified to focus 
only on identification of appropriate new assessment instruments through a pilot test 
of the instruments. The revised project objectives were therefore as follows; 

1. To pilot test promising candidate standardized instruments for potential addition to 
the Bureau’s list of standardized instruments allowable for reporting adult student 
learning gain data; 

2. To provide training, support, technical assistance, and capacity-building for use of 
CASAS by ABLE providers; and 

3. To coordinate implementation of training and technical assistance with the 
Training Module Project. 

Project Time Frame 

Project staff and Bureau of ABLE staff planned the pilot test in August 1999. 
Agencies were informed of the pilot and invited to participate in Fall 1999. Agencies 
were notified of their acceptance into the pilot in November 1999. 

The pilot continued through Summer 2000, with project staff providing 
communication, support, and technical assistance to participating agencies. The 
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results of the pilot test were analyzed in Summer 2000. Bureau of ABLE staff and 
the project staff consulted and agreed that the three instruments tested through the 

I 

pilot were useful for and useable by a variety of ABLE agencies for a variety of 
purposes, including reporting learner gain data. The Bureau of ABLE decided to add 
the instruments to their list of standardized assessment tools accepted for reporting 
adult student learning gain data. Project staff conducted necessary follow-up 
activities (e.g., developing draft educational gain standards, working with ABLE Net 
to modify the Literacy Pro data management software to include built-in 
specifications on the new instruments) in August and September 2000. 

Planning oP99 - ’00 CASAS training and capacity-building activities 
occurred in late ’99 and early ’00. Five CASAS training sessions were completed 
throughout Pennsylvania in Spring and Summer 2000. Support and technical 
assistance for ABLE-funded programs interested in CASAS or using CASAS was 
ongoing through the ’99 - ’00 program year. 

Project Staff and Key Personnel 

Carol Shefrin was the Project Director. She has over 15 years of experience 
in directing adult programs for the Tuscarora Intermediate Unit and developing and 
implementing special projects. She has directed large-scale special projects focusing 
on developing and implementing statewide training initiatives, including family 
literacy training and the Training Module Project. Other special projects she has 
directed have focused on research and development of assessment tools and practices. 
She co-authored, along with Lori Forlizzi and Barbara Van Horn, the “Assessment for 
Adult Learners” training module being delivered throughout the state as part of the 
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ABLE-funded Training Module Project. She directed all project activitie.s. oversaw 
fiscal management of the project, and assisted in planning the pilot test in conjunction 
with the Bureau of ABLE. 

The project's Training Coordinator was Dehra Shafer. She has man\- years of 
experience in adult education program and conference planning and has coordinated 
the Training Module Project. For the last two years, she has coordinated all of the 
ABLE-funded special project training initiatives (with the exception of family literacy 
training) that have been administered through the AEJTC. As part of the current 
project, she coordinated planning and implementation of the CASAS training tlirough 
the Training Module Project. 

Lori Forlizzi was Trainer/Training Developer for the project. She has four 
vears of experience in developing training on assessment topics and pro\ iding 
statewide training and technical assistance on assessment issues to ABLE-funded 
programs statewide. She is the lead assessment trainer for the Training Module 
Project, and authored or co-authored several of the assessment modules delivered 
under the auspices of that project, including “Assessment for Adult Learners,” 
“Hands-on Session with the TABE,” and “Hands-on Session with the BEST.” She is 
the also the lead state-level trainer for CASAS in Pennsylvania, and is the Assessment 
Strand Lead Trainer for the EQUAL initiative. She has provided consultation on 
assessment issues to the Bureau of ABLE. She was the lead researcher on the “ABLE 
Assessment Practices” project. She planned and coordinated the pilot test and related 
activities, including communication with and provision of technical assistance to the 
pilot sites. She assisted in planning the statewide CASAS training offerings. She 
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facilitated three of the CASAS training sessions offered as part of the project and 
provided information and technical assistance on CASAS to ABLE-funded agencies. 

Audience for this Report 

The audience for this report primarily includes staff of the ABLE Bureau and 
staff of ABLE-funded programs. 

Project Dissemination 

Copies of this report will be fded permanently at the following locations: 

1) Pennsylvania Department of Education. Bureau of Adult Basic and Literacy- 
Education 

333 Market Street, 12’*’ Floor 
Harrisburg, PA 17126-0333 

2) AdvancE State Literacy Resource Center 
333 Market Street, 1 1’*’ Floor 
Harrisburg, PA 17126-0333 

Statement of the Problem 

“Assessment Tools for Adult Education" thus continued, coordinated, and 
expanded two of the ABLE-funded assessment training and support initiatives 
operating in Pennsylvania during previous program years. The goal of the project 
remained to increase opportunities for training and technical assistance on assessment 
issues to staff of ABLE-funded programs in order to help them engage in continuous 
quality improvement and meet the requirements of the Workforce Investment Act. 
The focus of the project remained to provide training and support to staff of ABLE- 
funded programs to help them utilize assessment tools and procedures to effectively 
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document learning gains of the adult students they serve. The project 1) pilot-tested 
candidate assessment instruments for possible adoption by the Bureau of ABLE for 
reporting of learner gains by its liinded programs; 2) expanded training opportunities 
for CASAS through the Training Module Project; and 3) continued development of 
state-based CASAS trainers through the Training Module Project. 

Project Goals and Objectives 
The revised project objectives were as follows: 

1. To pilot test promising candidate standardized instruments for potential addition to 
the Bureau’s list of standardized instruments allowable for reporting adult student 
learning gain data; 

2. To provide training, support, technical assistance, and capacity-building for use of 
CASAS by ABLE providers; and 

3. To coordinate implementation of training and technical assistance with the 
Training Module Project. 

Procedures Employed 

“Assessment Tools for Adult Education” was conducted as two separate mini- 
projects: one pursuing the pilot test of candidate standardized instruments, the other 
the continuation expansion of statewide CASAS training through the Training 
Module Project. The procedures for each mini-project are detailed under the 
appropriate objective in the “Objectives Achieved ’ section below. 




15 



Assessment Tools Final Report p. 1 1 



Objectives Achieved 



Objective 1: To pilot test promising candidate standardized instruments for 
potential addition to the Bureau’s list of standardized instruments allowable for 
reporting adult student learning gain data. 

In August, 1999, the project staff met with Cheryl Keenan, Director of the 
Bureau of ABLE, and decided to devote the project year to a pilot-test of four 
candidate instruments: SLEP, Woodcock Reading Mastery Tests-Revised, Wonderlic 
Basic Skills Test, and Work Keys assessments. Two purposes of the pilot test were 
established: 

1 . To assess the appropriateness of each instrument for use with targeted populations 
of learners in ABLE-funded agencies, and 

2. To collect pre- and post-test data that would allow standards for learning gains to 
be established, should the pilot test be successful and the instruments be adopted. 

In August, 1999, the Trainer/Training Developer drafted an agency 
application form for the pilot test. The form was finalized in late August after input 
by the Project Director and Bureau staff (see Appendix A). It was distributed to 
agencies in September as an attachment to the ’99 - ’00 ABLE Assessment Policy, 
and as a handout at the EQUAL regional kickoff sessions in September. The 
application form, to be submitted to the Bureau no later than September jO, 1999, 
described the purposes of the pilot test; gave a brief description of each of the four 
instruments; and asked agencies, for each instrument they wished to pilot test, to 
indicate whether they a) had used the instrument prior to the project year and b) 
currently had someone on stalfwho had received training on the instiument. The 
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Trainer/Training developer was listed on the form as the contact for questions about 
the instruments and the pilot test. During September and October, she fielded related 
questions from interested agencies. 

In late October, the Bureau forw'arded all submitted applications to the Project 
Director and Trainer/Training Developer. They worked with the Bureau of ABLE to 
identify the pilot sites. No agencies volunteered to pilot test Work Keys, and so it 
was dropped from the pilot. 

The project team identified the following criteria for acceptance into the pilot: 

* All SLEP and Wonderlic Basic Skills Test applicants would be accepted, as both 
instruments were straightforward to administer and interpret. 

* Pilot test applicants for the Woodcock Reading Mastery Tests-Revised would be 
accepted if the agency had previously used the tests or had someone on staff that had 
previously received training on the tests. This requirement was set for two reasons. 
First, the testing process utilized in standardized administration of these tests is 
individualized and requires the test administrator to engage in on-line decision- 
making. In addition, the project team felt that training on the instrument could not be 
developed and implemented widely before the start of the pilot test. Thus, the team 
felt that restricting acceptance to those who had training on or experience with the 
tests was warranted. 

In early November, 1999, letters notifying each applicant agency of its status 
with the pilot (accepted/not accepted into the pilot) were mailed from the Bureau of 
ABLE. Acceptance letters indicated that while test materials and training on use of 
the instruments would not be available as part of the pilot, technical assistance and 
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support was a\ailahle to each participating agency from the Trainer/Training 
Developer. Six agencies applied to pilot the Wonderlic Basic Skills Test and all were 
accepted into the pilot. Seven agencies applied to pilot test the SLEP and all were 
accepted. Nine agencies applied to pilot the Woodcock Reading Mastery Tests- 
Revised; six were accepted into the pilot. The three agencies not accepted into the 
Woodcock pilot did not have training or experience with the instrument. 

Three agencies dropped out of the pilot test of the Wonderlic during the 
program year. In the case of one program, an arrangement that would have made it 
feasible to use the instrument fell through. In the case of the second program, an 
agency-wide reorganization had occurred; this, coupled with a staff shortage, made it 
impossible for the agency to continue to use the instrument. In the case of the third 
program, the agency decided that the instrument did not meet the needs of the 
population. One agency dropped out of the pilot test of the SLEP because the test 
materials w'ere not provided as part of the pilot. Thus, the final number of agencies 
piloting each of the instruments was as follows: 

SLEP; 6 

Wonderlic Basic Skills Test; 3 

Woodcock Reading Mastery Tests-Revised; 5 

A list of the agencies participating in the pilot is included in Appendix B. 

In late November and December 1999, the Trainer/Training Developer 
worked with the staff of ABLE Net to inform pilot agencies of how the pilot test data 
would be collected through the Literacy Pro (LitPro) software. In November, the 
Trainer/Training drafted and mailed letters to the pilot agencies that summarized the 
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data that could be collected and reported for each instrument, including the allowable 
subtests/subtest combinations, types of scores to be reported (e.g., raw. scaled, grade 
level equivalent), and how learner gain would be calculated (i.e., through scaled 
scores). Linda Hinman from ABLE Net reviewed these letters before they were sent 
to ensure that any LitPro needs were covered. In December, ABLE Net staff drafted 
and mailed, to participating agencies, directions for modifying LitPro to enter the data 
from the pilot assessments. The Trainer/Training Developer reviewed these 
directions before they were mailed to the pilot sites. 

In March 2000 (once programs would presumably have had a chance to begin 
using the tests) and July 2000 (once assessment data for the program year had been 
submitted), the Trainer/Training Developer contacted each pilot site to check on their 
progress and to determine if there were any problems or needs to be addressed. She 
also responded to requests for assistance generated by the pilot agencies. There were 
very few of these requests, and generally they related to how to obtain and use the 
tests (i.e., requests for contact information for test publisher, one question about how 
many/which alternative forms to purchase and use, and one question on how to use 
norms tables). 

In both March and July, the Trainer/Training Developer prepared open-ended 
interview questions and conducted telephone interviews with the sites. Thirteen of 
the fourteen pilot sites participated in the March interview (one Woodcock site did 
not participate in the interview). The March interview included the following 
questions; 

* Have you started using the instrument? 



19 



Assessment Tools Final Report p. 



* What programs arc you using the instrument in? What types of students are you 
using it with? 

* How many students have taken a pre-test with this instrument in this fiscal year? 

* How many students have taken a post-test with this instrument in this fiscal year? 

* Are students making gains from pre- to post-test? What do the gains look like? 

* Who is administering the instrument - teachers, counselors, test administrators, 
others? Are they having any difficulties administering the instrument? 

* Who is scoring/interpreting the results of the instrument? Are they having any 
difficulties with this? 

* Are you using the test results for puiposes other than tracking/reporting learner 
gains (e.g., planning instruction)? If so, what are those other purposes? 

* Is there any assistance I can give you or any questions I can answer at this time? 

All pilot sites participated in the July interview. The July interview included the 
following questions: 

* Are you administering the instrument to the same population? Describe any 
changes in the target population. 

* Which subtests are you administering to your students? Which are you reporting to 
LitPro? How do you determine which subtests to administer? Which to report? 

* How many students have taken a pre-test with this instrument this program year? 

* How many students have taken a post-test with this instrument this program year? 

* Are students making gains from pre-to post-test? If so, what is the average gain? 
How' many instructional hours usually occur between pre- and post-test? 
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* Do any gains you are seeing with the instrument correlate with the progress you 
think your students are making (based on your "gut" reaction or other sources of 
information)? 

* Are other or different staff now administering, scoring, or interpreting results of the 
instrument? Have any problems cropped up for anyone using the instrument? 

* What purposes are you currently using the instrument for? 

* Do you have any questions or need for assistance that I could help you with? 

* Would you say that the instrument has been a successful tool for your program? 
Why or why not? 

* Would you like to continue using the instrument in your program? W'hy? For what 
purposes? 

During both interviews, agencies were encouraged to volunteer any other helpful 
information and observations outside of these questions. 

The following paragraphs summarize the findings gleaned from the agency 
interviews. At the time of the interviews, sites were unsure of the average gains that 
their learners were making, or of the number of instructional hours attained. 

The five sites that piloted the Woodcock were, for the most part, using it with 
beginning level literacy learners in a one-to-one tutoring setting (however, one site 
also provided small group and classroom instruction; a second site administered it to 
learners reading up to the 7'*^ grade level). One site was using the instrument with 
developmentally disabled learners. Two sites reported pre-testing very small 
numbers of learners (5 each) and had not collected any post-test data by the time of 
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the July interview. The third site reported about 40 pre/post-test matches; the fourth 
about 70; and the fifth was unsure. 

All three Wonderlic pilot sites used the instrument with ABE/GED level 
learners in work preparation or job training classes; the learners at one site were 
adults with significant and chronic mental illness. One site reported five pre/post-test 
matches; the second. 51 pre/post-test matches; and the third was unsure of the number 
of students that had been pre and post-tested. 

The six sites piloting the SEEP served ESL students from a variety of 
countries of origin, mostly intermediate and advanced level students. While one site 
also used the instrument with beginning level ESL learners, other sites reported that 
their teachers felt that this would be an inappropriate instrument to use with 
beginning level ESL learners. These sites varied in the numbers of students that they 
reported having SLEP pre- and post-test matches. The numbers of students pre- and 
post-tested by the six SLEP pilot sites are as follows: 51; 10; 265; 0; 17; 23. 

Woodcock sites reported using the instrument for diagnosing learner needs, 
placing learners into instructional groups, and selecting instructional approaches and 
materials in addition to reporting gain scores to the Bureau of ABLE via LitPro. This 
is not surprising, as this instrument does provide rich diagnostic information on 
individual learners. Three SLEP sites reported using the test results for class or level 
placement; three reporting using it to try to glean information for instruction (but one 
reported finding that difficult in light of the information yielded by the tool). With the 
Wonderlic. two of the sites reported using the assessment results for class placement 
or for identifying learners who may have difficulties in an available level of classes. 
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Unlike the Woodcock, the SLEP and Wonderlic are not designed to provide detailed 
diagnostic information on learning needs. 

Most Woodcock sites reported administering and reporting scores on all of the 
available subtests. They chose to do so, again, because they found the diagnostic 
information provided by a combination of the subtests extremely valuable. Most 
SLEP sites stated that they administered the individual subtests and reported the 
individual subtest scores. Most SLEP sites felt that the individual subtest scores 
provided more useful information about student progress than the overall total score. 
Reports from the Wonderlic sites indicated no clear pattern in what they chose to 
administer and report. 

The pilot sites reported that a variety of types of staff administered, scored, 
and interpreted the instruments with virtually no problems. The types of staff who 
administered the tests included program directors, program coordinators, tutor/student 
coordinators, tutor trainers, office administrators, assessment specialists, teachers, an 
instructional assistant, an outreach coordinator, and a rehab coordinator. Only one 
agency (piloting the Woodcock) reported that the person administering the tests 
needed some assistance in getting started. 

Woodcock sites reported that they felt that the instrument was successful for 
working with very beginning level literacy learners, as a tool tor providing 
information useful in making diagnostic, placement, and instructional decisions. The 
site working with developmentally disabled learners was less enthusiastic; this site 
reported that the tool had worked better than the TABE with some (but not all) of 
their learners. All sites reported that they would like to continue to use the instrument 
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with at least some of their learners. Two out of the three Wonderlic sites felt that the 
tool had been successful in their agencies. They cited its brevity and abilit>- to 
provide the type of information they need for a broad assessment of skills and learner 
gain data reporting. The third site was positive, but wanted to utilize the tool more 
before committing to continued use. With one exception, SLEP pilot sites indicated 
that they believed the SLEP had been a useful tool for their intermediate and 
advanced level students; they cited its simplicity and noted that it filled a need for a 
standardized reporting tool for ESL students at this level. Five of the six sites 
indicated that they would most likely continue to use the instrument for reporting; 
sites felt that with long-term use of the instrument, they could get a better sense of 
what the test results meant in terms of instructional planning. The dissenting site felt 
that the tool did not adequately match their curriculum, and felt that time it took to 
administer the test was too long. They were unsure as to whether they would 
continue using SLEP. 

The pilot sites agreed that the Wonderlic and SLEP are simple enough to be 
picked up and used immediately by typical staff in ABLE programs. Those sites 
piloting the Woodcock felt that training or some sort of introductory overview of how 
to use the tests would be nice but not necessary. This is a bit surprising considering 
the nature of the Woodcock, and this finding should be interpreted cautiously as each 
of these pilot sites had some prior experience with or training in how to use the 
instrument. 

In summarv, the feedback from the pilot sites indicated that the instruments 
were usable by and beneficial to ABLE programs. Based on the these results from the 
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pilot test, the Bureau of ABLE decided during late Summer ’00 to add the three 
instruments to its list of instruments accepted for reporting adult learner educational 

J 

gains. Draft educational gain standards for each instrument would be developed 
based on the pilot pre- and post-test data submitted by the pilot sites to the Bureau via 
LitPro in July ’00. 

This step was completed in August 2000. At that time. ABLE Net supplied to 
the Trainer/Training Developer all available LitPro assessment data from the pilot. 

The Trainer/Training Developer analyzed the data and developed draft educational 
gain standards for the instruments to be included in the ’00 - ’01 ABLE Program 
Performance Standards document (see Appendix C). It should be noted that not all of 
the pilot sites had provided complete pre- and post-test data on the pilot instruments 
by that time; in addition, the data from sites serving special populations — 
developmentally disabled and mentally ill students - were reviewed but not used to 
calculate the draft educational gain standards. 

During Summer of ’00, the Trainer.Training Developer also worked with 
Linda Hinman of ABLE Net and Bill Murphy of the Bureau of ABLE to specify 
allowable subtests/subtest combinations and scale score ranges for the ’00 - ’01 
version of LitPro. This step was necessary for specifications on the new instruments 
to be “hard-wired” into LitPro. Since the pilot instruments had not been widely used 
by ABLE agencies prior to the pilot year, during the pilot year sites had to manually 
specify pilot instrument subtest names in LitPro. There were no built-in restrictions 
or edit checks on scale score entry. 
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Final project activities rclatecl to the pilot test were completed in August and 
September 00. I'he Trainer/ Training Developer assisted the Bureau in developing 
policy guidance on the three pilot instruments (regarding the appropriate populations 
for each instrument). She also developed reference values for placing learners into 
the National Reporting System Educational Functioning Levels based on scaled 
scores and worked with ABLE Net and the Bureau of ABLE to add these to 
Pennsylvania’s Educational Functioning Level Reference Chart in the 00 — 01 
ABLE data collection instructions manual published by ABLE Net. 

Objective 2: To provide training, support, technical assistance, and capacity- 
building for use of CASAS by ABLE providers. 

Preliminaiy planning for the ’99 - ’00 project year was conducted by the 
Training Coordinator, the Trainer/Training Developer and Linda Taylor, 
Pennsylvania’s point of contact with the national office of CASAS, in Fall 99. In 
January 2000, the Training Coordinator and the Trainer/Training Developer 
conducted a telephone survey of ABLE programs to identify CASAS training needs. 
They contacted 24 ABLE-funded agencies that had purchased CASAS materials (the 
list of agencies was supplied by CASAS) to determine 1) whether the agency was 
using the Employability Competency and/or Life Skills systems, and 2) what, if any, 
CASAS training, technical assistance, or support needs the agency had. Through 
these telephone interviews, the project team identified 16 agencies using these 
CASAS systems, and determined the need for three initial level trainings (east, 
central, and west) and 2 advanced level trainings (east and west). 
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The following CASAS training sessions were completed during the '99 - '00 
project year: 

I 

* Initial Implementation Training, 4t2SI00, Pittsburgh: served 12 agencies (25 
participants) 

* Initial Implementation Training, 7/19/00, Philadelphia: served 12 agencies (21 
participants) 

* Initial Implementation Training, 7/20/00, Philadelphia; served 14 agencies (17 
participants) - this was an additional (unplanned) session set up in the Philadelphia 
area in order to accommodate the overflow of registrations for the 7/19 session. 

* Advanced Training, 5/16/00, Harrisburg: served 8 agencies (2 j individuals) 

* Advanced Training. 5/18/00, New Castle: served 9 agencies (14 individuals). 

An additional Initial Training was scheduled, as a result of the telephone 
survey, for 9/14/00 in Wilkes Barre; this session was postponed until the next project 
year due to low registration numbers, but had been rescheduled for 10/19/00 by the 
end of the project. 

The project also built capacity for in-state CASAS training and technical 
support for CASAS. CASAS requires that candidate state-level trainers-in-training 
complete a training apprenticeship in order to be certified as CASAS state-level 
trainers. By the ’99 - ’00 project year, Trainer/Training Developer had completed the 
certification process and was certified to facilitate training in the Employability 
Competency and Life Skills Systems. She facilitated the 4/28, 7/19, and 7/20 Initial 
Training sessions. Four additional Pennsylvania-based candidate trainers-in-training 
(Ellen McDevitt, Suzanne Webster, Susan Finn Miller, and Linda Wolfson) were 
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advanced in the apprenticeship requirements by observing and/or assisting with 
components of these sessions. Ardis Breslauer. a CASAS national-level trainer, 
facilitated the two Advanced Training sessions. 

During the project year, the Trainer/Training Developer also provided 
technical assistance and support on issues related to CASAS to ABLE-funded 
agencies throughout the state. She provided information to interested agencies, 
helped to promote the training by word-of-mouth to interested agencies, and provided 
assistance regarding questions about selecting, purchasing, and implementing the 
CASAS Employability Competency and Life Skills systems. 

The Training Coordinator organized and led a panel discussion session, 
entitled “Is CASAS for You?.^’ at the February 2000 PAACE Midwinter Conference. 
The presentation included an introduction to the CASAS Employability Competency 
and Life Skills systems and possible applications of the systems. Four panelists from 
ABLE-funded agencies that were using these CASAS systems then provided an 
overview of how CASAS was being used in their agencies, why and how the decision 
was made to use CASAS, and the successes and challenges they had faced in its 
implementation. The panelists included the Training Coordinator, the 
Trainer/Training Developer, Susan Finn Miller of Lancaster-Lebanon lU 13, Barb 
Noel of Franklin County Literacy Council, Peggie Rood of Luzerne County 
Community College’s ALTA Program, and Michael Westover of Catholic Charities. 
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Objective 3: To coordinate implementation of training and technical assistance 
with the Training Module Project. 

I 

The goals of the Training Module Project are to develop or adopt quality 
training on topics of interest to ABLE*funded agencies; to develop and maintain a 
delivery system for the training in conjunction with the ABLE-funded regional 
Professional Development Centers; and to train, monitor, and support regional 
trainers for each module topic. A clear need had been established to provide ABLE- 
funded agencies with information about and training on the CASAS Employability 
Competency and Life Skills systems, particularly as assessment tools that provide an 
option for assessing ESL learners at a wide range of ability levels, through the 
‘^ABLE Assessment Practices” project conducted in the ’98 - ’99 program year. All 
training and technical assistance activities around CASAS were smoothly integrated 
with ongoing Training Module Project activities. The Training Coordinator, assisted 
by the AEJTC’s Training Team and the Trainer/Training Developer, planned and 
organized all five CASAS training sessions in the 99 — 00 program year in 
conjunction with the needs of aigencies and Professional Development Centers in each 
region. CASAS trainers-in-training were observed, supported, and assisted by the 

Trainer/Training Developer. 

Evaluation Techniques, Instruments, and Results 
Throughout the pilot test, the project team worked closely with the Bureau, 
ABLE Net, and the pilot sites to monitor progress toward meeting the objective. The 
design of the pilot test ensured that the field contributed to identifying standardized 
assessment instruments that could be used to document the educational gains of adult 
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learners in ABLE populations. The open-ended interviews provided opportunity for 
pilot sites to make suggestions and provide feedback regarding the feasibility of the 
instruments and the pilot test itself. 

The provision of training, technical assistance, and support around CASAS 
throughout the project benefited from the evaluation processes previously set up as an 
ongoing part of the Training Module Project. The Module Project evaluation 
protocol solicits evaluation information from trainers (via contact with the lead 
trainer), Professional Development Center staff (via contact with the Training 
Coordinator), and training participants (via an end-of training evaluation form and 
follow-up contact provided by the trainer). Informal correspondence with the 
Professional Development Centers and CASAS trainers and feedback from the 
training participants captured via the end-of-training evaluation form indicated that 
the CASAS training offerings were of high quality and met the needs of participating 
ABLE agencies. 

Procedures for Dissemination of the Findings and Products 
This final report will be disseminated through the AEJTC, the Bureau of 
Adult Basic and Literacy Education, ABLE’s regional Professional Development 
Centers, and the AdvancE State Literacy Resource Center. 

Conclusions and Recommendations 

Through this project, three additional assessment tools that are appropriate for 
assessing learning gains of adult students in ABLE-funded programs were identified 
and demonstrated to be usable by and useful for a variety of ABLE-funded agencies. 
These were added to the Bureau of ABLE’s current list of standardized assessment 
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tools accepted for reporting learning gains of adult students in ABLE-funded 
programs. Results indicated that building awareness of the instruments and training 
on the instruments would be useful to ABLE-funded agencies. There may be a 
particular need to provide some training around the Woodcock Reading Mastery 
Tests-Revised to those agencies that would like to use the instrument. 

The project also demonstrated a high interest among ABLE-funded agencies 
in the CASAS Employability Competency and Life Skills assessment systems. 
Regional trainings on CASAS were well-attended and well-received. Furthermore, 
the CASAS training was smoothly integrated with the training and technical support 
system set up through the Training Module Project. Both the Initial and Advanced 
trainings should continue to be made available to the staff of ABLE-funded programs. 
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BUREAU 

adult basic a 
litcract education 



gjj|(0 Trial Assessment Instrument Registration Form 
o7"«T“r Program Year 1999 -2000 

EouCAnpN f 

In program year 1999-2000, the Bureau of Adult Basic and Literacy Education invites funded 
agencies to register to pilot-test four standardized assessment instruments on a trial basis. The 
instruments are: 

□ The Woodcock Reading Mastery Test-Revised 

□ The Secondary Level English Proficiency Test 

□ The Wonderlic Basic Skills Test 

□ Work Keys assessments (delete pending relevant info, from Mike Snyder??) 

The purpose of this pilot-test is two-fold: 

1. To assess the appropriateness of the instruments for use with certain populations of learners 
in ABLE-funded agencies; 

2. To collect pre- and post-test data that would allow standards for learner gains to be 
establishsd should the pilot-test be successful and the instruments be adopted for reporting 
learner gains on a permanent basis. 

Agencies interested in registration for the pilot-test should complete the following and submit, no 

later than September 30, 1999, to: PA Department of Education 

Bureau of Adult Basic & Literacy Education 

333 Market Street, 12th Floor, 

Harrisburg, PA 17126-0333— fax: 717-783-0583 

Agencies will be contacted to inform them of their approval status. 




Indicate which of the four instruments you will be pilot-testing; for each, answer the 
accompanying questions regarding the instrument. 



Woodcock Reading Mastery Test-Revised 

Description; Individually-administered diagnostic instrument appropriate for 
measuring skills of adult beginning readers (word identification, word 
attack, word comprehension, and passage comprehension). 

Contract number(s) under which you will use the instrument 
□Y QN We have used this instrument prior to this fiscal year. 

Q QY On Someone on staff has received formal training on this instrument. 

ERIC 
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Secondary Level English Proficiency Test 

Description’ Measures secondary-level English language skills in listening 
comprehension and reading comprehension; may be group or individually administered. 

Contract number(s) under which you will use the instrument 

□Y QN We have used this instrument prior to this fiscal year. 

□Y QN Someone on staff has received form.al training on this instrument. 



Wonderlic Basic Skills Test 

Description: Measures basic reading and math skills of adult learners at the upper-ABE 
and GED skill range; results can be compared to the basic skills requirements of 
occupations listed in the U. S. Department of Labor’s “Dictionaiy of Occupational Titles 
Requires computer scoring with a disk provided by Wonderlic, Inc. (IBM or compatible 

computer). May be group or individually administered. 

Contract number(s) under which you will use the instrument 

□Y QN We have used this instrument prior to this fiscal year. 

□Y QN Someone on staff has received formal training on this instrument. 



Work Keys assessments ■ r ^ 

Description; Eight available instruments measure listening, reading for information, 
writing, applied mathematics, applied technology, locating information, observation, and 

teamwork. 

Contract number(s) under which you will use the instrument 
□Y QN We have used this instrument prior to this fiscal year. 

□Y QN Someone on staff has received formal training on this instrument. 



Questions regarding the registration process and assessment instruments shouid be directed to 
Lori Forlizzi 

South Central Professional Development Center 
Tuscarora Intermediate Unit 
Telephone (717) 248-4942 
email - forlizzi@sprynet.com 
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Appendix B: 

Agencies Participating in the Pilot Test 
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Agencies pilot testing the Woodcock Reading Mastery Tests-Rcviscd; 

Community Action, Inc., Punxsutawney 

Crawford County Literacy Council, Inc., Meadville 

Focus on Renewal Sto-Rox Neighborhood Corporation, McKees Rocks 

Programs Employing People, Philadelphia 

The Literacy Council of Mercer County, Greenville 



Agencies pilot testing the Wonderlic Basic Skills Test; 

Northwest Tri-County Intermediate Unit 5, Edinboro 
Stairways Behavioral Health, Erie 

Tri-County Opportunities Industrialization Center, Inc., Harrisburg 



Agencies pilot testing the Secondary Level English Proficiency Test; 

Bradford County Library (Bradford-Wyoming County Literacy Program), Troy 
Community Learning Center, Philadelphia 
Lancaster-Lebanon Intermediate Unit 13, Lancaster 
New World Association, Philadelphia 

Pittsburgh School District (Connelley Technical Institute), Pittsburgh 
Volunteer English Program in Chester County, West Chester 
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Appendix C: 

Draft Educational Gain Standards for the Pilot Instruments 
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Woodcock Reading Mastery Tests-Revised; an increase of 10 W (scale) score points 
on the Total Reading Cluster-Full Scale. 

Secondary Level English Proficiency Test; an increase of 2 scale score points on thd 
Reading Comprehension subtest, or 2 scale score points on the Listening 
Comprehension subtest. 

Wondeiiic Basic Skills Test: an increase of 25 total (scale) score points on the Verbal 
Skills subtest, or 25 total (scale) score points on the Quantitative Skills subtest. 



O 

ERIC 



38 






U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 

1 1 National Library of Educatmn (NLE) 

Educational Resources Information Center (ERIC) 

reproduction release 

(Specific Document) 

1. DOCUMENT IDENTIFICATION: 


<g) 


ERIC 




Assessment Tools for Adult Education 


Authoits): Carol Shefrin 


Corporate Source, xuscarora Intermediate Unit 11 


Publication Date; 

October, 2000 


II. REPRODUCTION RELEASE: 



In order to dissetninate as widely as possible timely and significant materials of intetest to the educational community, documents announced in the 
monthly abstract journal of the ERIC system, Resources in Education (RIE), are usually made available to users in microfiche, reproduced paper copy, 
and electronic media, and sold through the ERIC Document Reproduction Service (EORS). Credit is given to the source of each document, and, if 
reproduction release is granted, one of the following notices is aff^ed to the document. 



If permission is granted to reproduce and disseminate the identified document, please CHECK ONE of the following three options and sign at the bottom 
of the page. 




Level 1 



0 



Level 2A 



□ 



Level 2B 

t 

□ 



Chacfc hara for Level 1 releete. penntttna leproducdon 
andd iee efri n ebooinmi cn iS ch eof ctwr ERICarcMvai 
media (ea.. etectrenic) and paper copy. 



Check Im for Level 2A tWeese. pemilttng r 



and dieaeminstfon in (TdcraSche and in electronic medle 
for ERIC arcNvil eolecbon subeofoera only 



Cha^ hare for Level 2B release, penrttbng 
reproduction and dissemination In miuoftche only 



Documents wfl be proce ss ed as In dfo a te d prevMed reproductton qutfty permits. 

If permi sai on to fsptoduce is yanted. but no box is checked, documerrtswtt be processed at Level 1. 



Sign 

here,-* 

please 



1 hereby grant to the Educationai Resouroes Information Center (ERiC) nonexclusive permission to reproduce and disseminate this document 
as indicated above. Reproductidh from the ERIC microfiche or electronic media by persons other than ERIC employees end its system 
contractors requires penriisshnih^m the oopyrigM holder. Exception is rnade for non-prcdH reproduction by libraries end other service agencies 
to satisfy information needs of educators in response to discrete inquiries 

voJ /) L ^ 


c/. 


Printad Narne/Poskiorinitle: 

Cheryl L. Keenan, Director 


ckg-iUBfiu,^Ad(kass. Dept, of Education 

333 Market St., Harrisburg, PA 17126-0333 


T**^717-772-3737r**-717-783-5420 


E-Mea Address: 1 Dste: 

IstasiulatPstate .pa.us 6/22/01 



O 

ERIC 



(over) 




III. DOCUMENT AVAILABILITY INFORMATION (FROM NON-ERIC SOURCE): 

If permission to reproduce is not granted to ERIC, or, if you wish ERIC to cite the availability of the document from another source, please 
provide the following information regarding the availability of the document. (ERIC will not announce a document unless it is publicly 
available, and a dependable source can be specified. Contributors should also be aware that ERIC selection criteria are significantly more 
stringent for documents that cannot be made available through EDRS.) 



Publisher/Distributor: 



Address: 



Price: 



IV. REFERRAL OF ERIC TO COPYRIGHT/REPRODUCTION RIGHTS HOLDER: 

If the right to grant this reproduction release is held by someone other than the addressee, please provide the appropriate name and 
address: 




V. WHERE TO SEND THIS FORM: 



Send this form to the following ERIC Clearinghouse: 



However, if solicited by the ERIC Facility, or if making an unsolicited contribution to ERIC, return this form (and the document being 
contributed) to: 

ERIC Processing and Reference Facility 
1100 West Street, 2^ Floor 
Laurel, Maryland 20707-3598 

Telephone: 301-497-4080 
Toll Free: 800-799-3742 
FAX: 301-953-0263 
e-mail: erlcfac@ineted.gov 
^ • WWW: http://ericfac.plccardx8c.com 

EFF-088 (Rev. 9/97) 

PRFVir^l 1.^ VFP-^inMQ nc TUic cnoik4 aoc r\oor\\ cxc 



o 



